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Preface 


This report describes the structure and effectiveness of a NASA program that 
replicated a private sector research and development partnership model. The 
partnership is described as an "alliance" as it included a large number of participants 
organized into work-teams. 

The concept in adopting the alliance was very simple: to gain the same 
efficiencies in program management that are realized in private sector alliances, 
namely improved outcomes at a lower cost with better customer satisfaction. 

Private sector management concepts, however, rarely translate readily into 
public sector activities. Our system of government management reflects the broader 
ebb and flow of public policy regarding the Federal government role, and how 
bureaucracy provides for the public good. The introduction of alliance management 
models into the Federal government must deal with four broad trends in federal 
management philosophy. a 

o Scientific management with its focus on tight hierarchy, chains of 
command and specialization 

o War on waste with its emphasis on inspectors, auditors and reviewers 

o Watchful eye with its embrace of sunshine and openness 

o Liberation management with its cry to let the managers "manage" with 
some market pressure 

Alliance models cut across all four trends. Alliances contain a form of 
structure that provides for hierarchy and chains of command. Equally, they contain 
internal audit processes and constant peer review of both cost and output, invoking a 
level of knowledge above that of an outside auditor. While alliances can provide 
some form of liberation from traditional management processes, that liberation is 
offset with rules regarding joint decision-making. 

So if alliances contain all these elements, almost the best of all worlds, why are 
they not more readily used? Simply put, because they are customized business 
arrangements that satisfy a particular strategic or program need. And, the very 
nature of customization means that alliances must be carefully designed to be 
effective in meeting stakeholder expectations. Those expectations are measured by 
asking "participants" whether the alliance met its performance objectives, and how 
favorably that experience compared to other business vehicles. b 


a Paul Light, The Tides of Reform: Making Government Work 1945-1995, Yale University Press, 
New Haven, CN, 1997. 

b Accenture Consulting Study, Management Review, American Management Association, Page 7, 
January 2000. 
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Executive Summary 

Background 

This report describes the collaborative program model chosen to implement an 
aeronautics research and technology program from 1994 through 2001: the Advanced 
General Aviation Transport Experiments (AGATE) Program. The Program had one primary 
objective: to improve the ability of the General Aviation industry to adopt technology as a 
solution to fulfill public benefit objectives . The program was launched in response to the 
widening gap between the national need for improvement in the safety and performance of 
general aviation aircraft, and the inability of the General Aviation industry to adopt new 
technology for those improvements. 

Objective 

The primary objective of this report is to assess the program's ability to meet a 
combination of "effectiveness measures" from multiple stakeholders. The "effectiveness" of 
any model forms the foundation of legitimate questions for policy makers and professional 
federal managers. Within this context, it is the objective of this report to determine how such 
models can be assessed for their effectiveness by focusing on four questions: 

• What basic NASA R&D program model did AGATE follow? " 

• What tailoring was done for the particular AGATE program model? 

• What criteria were applied to assess the effectiveness of the tailored AGATE 

program model? 

• How well did the AGATE program perform relative to those criteria? 

Scope and Sources 

Information about the effectiveness of the AGATE program model were drawn from 
five sources: a) AGATE Closeout Reports authored for all six (6) of the "technical teams" 
covering the public and private members, b) interviews with a total of twenty-six (26) NASA 
program managers, AGATE private sector members, and independent policy analysts, 
c) results of four (4) "commercialization" performance surveys conducted from 1996 through 
2000, d) results of nine (9) performance audits undertaken by an independent auditor, and 
e) the results of one independent NASA OIG review of the program. 

AGATE Program Model 

AGATE was organized as a public /private partnership, focused on standards 
development. The partnership was "horizontal" in nature, with competitors collaborating to 
create "pre-competitive" standards. Public and private members joined into working teams 
that set strategy and selected specific research topics. The alliance was under the overall 
management of NASA as the "senior partner." 

The final integrated organizational structure was a hybrid of basic project command and 
control structures, with layers of working groups to coordinate decisions. NASA took the 
leadership role at each level, except for basic facilitation and business administration, while 
industry took a "membership" role at each level. The organization had six components: 

■ General Aviation Program Office (GAPO)- The NASA program office staffed by the 
Program manager, a deputy and secretary. GAPO was the overall "alliance" manager 
and had final authority to approve all plans. 

■ Executive Council- Composed of elected representatives from industry technical teams 
and designated government representatives. The Council's primary function was to 
agree on strategy and priorities. 

■ AGATE Management Team- Composed of the GAPO personnel and government 
technical specialists that led six technical working teams. The Management Team's 
primary function was to assure coordinated implementation of the annual plan. 

■ Systems Assurance- Led by a Government specialist and composed of industry members 
representing each of the technical working teams. The Systems Assurance team's primary 
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function was to support systems integration across technical teams and assure work was 
done to overall "configuration management." 

■ Technical Teams- Led by Government specialists and composed of industry and 
academic members. These were the basic organizational unit that prepared and executed 
annual technical plans to achieve the program's objectives. 

■ Facilitation/ Business Administration- Led by private sector managers, this function was 
delivered through by a non-profit private sector entity, and funded by both the public 
and private sector. The facilitation function acted as a good-faith intermediary between 
the public and private sector members. The business administration function provided 
eight basic services necessary for a partnership-based program. 


Private Sector 
Membership 


Members 

Of 

Executive Council 
Systems Assurance 
Technical Teams 


Levels of 
Membership 

• Principles 

• Associates 

• Supporting 


General Aviation Program Office 

• Program and alliance management 
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•Strategy and annual plans 


AGATE Management Team 

• Government only managers 

• Leaders of technical teams 

• Coordinates plan implementation and tactical tradeoffs, 
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Assessment Framework 

There were four levels of effectiveness that applied to the AGATE Program over its 
lifetime of planning and implementation: 1 

• Policy - Major principles that guide the justification all Federal programs. 

• Department /Agency Strategy - Strategic goals to be accomplished or supplemented by all 
Agency programs . 2 

• Large Organizational Unit - NASA Center (Langley, Ames, Glenn) level measures 
including their respective charters, and management system requirements. 

• Program - Measures specific to a specific program (e.g. aeronautics knowledge 
advancement). 

There were eight measures of effectiveness that were applied to the AGATE program . 3 4 5 
These measures were compiled from management initiatives launched over the past four 
administrations. 

• Performance- Creation of measurable goals to be achieved through program plans, 
organization structures, and management structures with high quality, on schedule and 
within budget. 

• Public Benefit without Private Preference - Creation of program outputs designed to 
generate benefits to the American public without subsidy to private participants or 
control of the outcome by a limited number of organizations. 

• Relevance- Definition of goals and objectives relevant to customer /user groups. 

• Speed- Generation of results quickly and in forms that can be readily adopted by 
customer /users. 

• Accountability- Creation of organizational structures with clear responsibility and 
monitoring systems tied to specific performance measurements. The structures and 
systems hold program and task level managers accountable for all Levels of objectives. 

• Cost- Reduction of costs in all categories of program content and management. 

• Flexibility - Use of legal authorities, organizational structures, and decision processes to 
respond to changing needs from policy, management, and research & development 
finding. 

• Institutional Development- Development of institutional capacity for further research 
and development support. Development of these capacities includes improved human 
capital, facilities, and new knowledge. 

The cumulative program effectiveness measures generated at different levels of 
oversight created a single model for the AGATE program effectiveness measurement. The 
format is designed to mirror the evolution in performance measurement summaries 
developed under the last two administrations . 4 5 


1 Derived from interviews with Bruce Holmes, Sam Morello, and David Trinkle (OMB) 

2 NASA Strategic Management Handbook, as revised NPG 1000.2, Washington, DC, 2000 

3 A Report to the 43 rd President and 107 th Congress: Transitioning to Performance-based Government, 
November 2000 

4 5th Year Report on Re-Inventing Government, Brooking Institution, Washington, D.C., 1998 

5 Program Assessment Rating Tool (PART) explanation, provided by David Trinkle to Paul Masson 
for interview. 
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MEASURES 

Qualitative 

Rating 

(Hi/Med/Low) 

Feedback from Participants of Effectiveness 
regarding Policy, Strategic, Organizational 
Unit and Program/Project Levels 

Performance 



Relevance 



Benefit w/out 
Preference 



Accountability 



Cost 



Speed 



Flexibility 



Institutional 




Summary of Effectiveness Conclusions 

The participants rated AGATE as achieving its primary objectives and rating well on 
effectiveness in most areas. There was a clear pattern of tradeoff, with high measures for 
relevance, cost, speed and public benefit, but lower measures for institutional fit and 
flexibility at dealing with the larger NASA organizational structure. This tradeoff reflects 
similar patterns of tradeoff found in private sector surveys of alliances and partnerships. 

The effectiveness measures were assessed against the program's stated goals. There 
were several "stretch" goals, in particular integrated flight tests that would represent the 
evolution of the technology integration. The area of stretch goals is where the participants 
rated AGATE lower. Most participants recognized that the lack of integration was a tradeoff 
for a desired "decentralized" approach permitting work teams to operate independently on 
selecting specific technology targets. 

The business managers at NASA rated the program medium to low, focusing on the 
difficulty of implementing a program structure that contained more tailoring than most 
NASA LaRC programs. 

The summary of effectiveness responses is provided in the following table. 
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Summary of Effectiveness Measure Feedback 


MEASURES 

Qualitative 

Rating 

Effectiveness at Policy, Strategic, Organizational 
Unit and Program/Project Levels 

Performance 

High 

Performance was achieved at all levels. The 
decentralized structure of the alliance made it difficult 
to achieve a “stretch goal” of integrated flight tests. 

Relevance 

High 

The relevance of program/project outputs scored high 
by the industry end-users and the FAA partners. 
Disagreement over relevance was limited to specific 
technology targets rather than the overall 
program/project approach. 

Benefit w/out 
Preference 

High 

Public benefit without private preference scored high 
with all participants due to the focus on creation of 
technology standards rather than specific technologies. 

Accountability 

Medium 

Accountability was achieved at the Policy, Strategic 
Levels. Financial accountability was achieved at the 
Center and Program level, but technical accountability 
was reported as inconsistent for some of the work- 
teams at the program/project level. This inconsistency 
at the program/project level created questions of 
accountability at the Center level. 

Cost 

High 

The cost effectiveness was considered high at all 
levels. The program secured cost sharing from private 
sector partners and organized highly effective 
leveraging agreements with other NASA programs 
(SBIR) and the FAA. 

Speed 

High 

The speed of tasking and undertaking work was 
considered high, when compared to other Federal 
program/project forms. The speed of program 
“formulation” was the only negative rating. This was 
attributed to the need to secure “agreement” with 
partners before proceeding. 

Flexibility 

Medium 

Flexibility was achieved at the Policy, Strategic and 
Program/Project levels. Changes in national priorities, 
NASA’s strategy and project targets were 
incorporated into the program. This flexibility, 
however, caused difficulties at the Center level need 
for consistent program/project management. 

Institutional 

Low 

The program did not work within Center level 
institutional structures nor build further research and 
technology competencies. Center level business 
managers found the program difficult to implement 
and potentially risky due its particular form of 
accountability management. 
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BACKGROUND 


Few things can better serve managers undertaking leadership roles in large, complex, 
technology-based organizations than knowing how other such organizations have coped 
with the problems that must be solved to achieve success. 6 

AGATE Program: A Collaboration with the General Aviation Industry 

This report describes the collaborative program model chosen to implement an 
aeronautics research and technology program from 1994 through 2001: the Advanced 
General Aviation Transport Experiments (AGATE) Program. The Program had one primary 
objective: to improve the ability of the General Aviation industry to adopt technology as a 
solution to fulfill public benefit objectives . The program was launched in response to the 
widening gap between the national need for improvement in the safety and performance of 
general aviation aircraft, and the inability of the General Aviation industry to adopt new 
technology for those improvements. 

Collaboration Embedded Within NASA’s System 

The range of NASA mission's drives a fundamental need for the Agency to engage in 
collaborations of all scales and types. A large academic research community and cadre of 
contractors has become a permanent part of the Agency's resource pool. A wide range of 
contracting and cooperative agreement authorities have been standardized along with 
selected use of NASA's Space Act Authority. This accumulation of effort and tools 
represents the evolution of a "collaboration system" reflected in different program models 
available to NASA managers. 

Evolution of the Collaboration Culture 

The genesis of NASA's collaboration culture is based in the operating practices of 
NACA, the aeronautics agency that absorbed multiple rocket programs to create the current 
day NASA. The aeronautics agency held annual "industry engineering conferences" 
meetings from 1931 to 1939 with industry groups to identify commonly needed aerodynamic 
design requirements. 7 An extension of this "common approach" led to the practice of 
industry collaborative use of NASA's wind tunnel system. 

This foundation was expanded with the space program to add the "contractor 
model" and its supplemental network of academic research. This provided the agency with 
a steady flow of internally and externally generated research, integrated by contractors into 
programs directed by NASA civil servants. 


6 Arnold Levine. Managing NASA in the Apollo ERA. NASA Special Publication # 4102, 
National Aeronautics and Space Administration, Washington, DC, July 1972. 

7 Roger Bilstein. Orders of Magnitude: A History of NACA and NASA: 1915 to 1990. National 
Aeronautics and Space Administration, Washington, DC, 1989. 
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Evolved Structure of Collaborative Models 

The evolution of the collaborative culture has continued in recent years, with new 
forms of collaboration designed to copy private sector management methods. When 
combined with NASA's original collaborative culture, it provides a basic array of options for 
managers when faced with collaboration requirements: 

Table 1 

NASA Collaborative Approaches 


Collaborative Category 

Features of the Collaboration 

Research exchanges 

Informal cooperation, often with academic community, 
resulting in co-authored papers 

Joint data generation 

Formal agreements for data generation, often with 
aeronautics companies, using Wind Tunnel agreements 

Contractor arrangements 

Formal goods and services acquisition, structured with a 
high level of personal collaboration among participants 
generating a "team" 

Program/Project Partnerships & 
Alliances 

Formal structuring of cost and results shared projects; 
resulting in joint project plans, controls and output 
sharing 

Privatization 

Transfer of assets to private sector control, in return for 
ongoing Federal use, resulting in a sharing of operations 
vs. asset ownership risk 

Outsourcing 

Transfer of basic functions to contractor outsourcing, but 
maintaining a close working relationship between the core 
NASA managers and their outsourced managers 


General Aviation Industry: A Slow Loss of R&D Capabilities 

General aviation refers to the smaller category of aircraft used for personal, small 
package, and business transportation. The general aviation industry is composed of 
companies that manufacture the aircraft, schools that train the pilots, and airport service 
operators known as "fixed-base-operators." The entire industry is regulated by the Federal 
Aviation Administration (FAA) to assure safety of planes, pilots and their operation in the 
national air space. The process of regulating the safety of individual aircraft takes the form 
of a detailed review of any new equipment or planes proposed for sale by a general aviation 
manufacturer. This review process results in a "certification" of the plane's safety and the 
manufacturer's ability to begin full-scale production for sales. 

The lack of technology adoption for new general aviation product improvements was 
caused by a series of factors. The high costs of product liability coupled with uncertain 
certification costs for new technology caused many companies to slow the rate of product 
improvements. The advancement of information technology into large aircraft could not 
migrate into the smaller aircraft for lack standards. While many small companies developed 
new technologies, few had the resources to overcome both the certification process and 
assure standards compatibility with other suppliers. 

These conditions combined in the industry to divert resources away from research 
and development. As the internal technical capabilities of the industry shrunk, their ability 
to readily adopt externally generated technologies declined. Industry leaders recognized the 
absence of technology-based innovation, but no single company or trade association could 
muster the support to launch an industry-wide effort to comprehensively deal with the 
issues of certification and industry standards. 
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AGATE Program Formulation: Rebuilding R&D Capabilities and Links 

The AGATE Program was launched in 1994 based on seven (7) recommendations 
generated by a General Aviation Task Force composed of government, industry and 
academia. 

Four recommendations focused on where to revitalize the research and development 
capabilities of the industry: 

• Propulsion, Noise and Emissions- Evaluate technologies that will permit the aircraft to 
operate quieter, with fewer emissions, using and using less fuel while meeting 
environmental standards throughout the world. 

• Aeronautics Systems- Introduce new technology in the cockpit to enhance pilot training and 
improve aviation safety. 

• Structures and Materials- Transfer available NASA information on advanced metallurgic 
and composite materials to the private sector. 

• Aerodynamics- Introduce technology for greater efficiencies in vehicle speed, carrying 
capacity, and fuel consumption. 

Two recommendations focused on re-starting general aviation research at NASA, 
which had been terminated in the mid-1980's. The final recommendation focused on 
considering use of collaborative models to implement the program. 

Program Management Options 

The program manager considered five fundamentally different ways of designing the 
AGATE Program management approach, given the practices and authorities that existed at 
NASA between 1992 and 1995. 

In-House Model- NASA in-house research, technical reports & demonstrations - Develop and 
execute a research and demonstration program focusing on in-house research and 
technology competencies. The research is organized by topical area of expertise. Each 
expertise area leader chooses the mix of civil servant and contractor expertise to complete 
and deliver research results. The results are distributed publicly via papers and symposia. 
Competitive Procurement Model- Procure a demonstration plane- Develop and demonstrate an 
advanced aircraft working with a contractor secured via a competitive procurement. The 
contractor will disseminate knowledge through demonstration and public distribution of 
papers. 

University Research Model- University led, industry research, technical reports- Develop and 
demonstrate individual technologies working with broad based industry teams working 
under university leadership. The teams are organized by research topic and disseminate 
knowledge through joint project work and public distribution of papers at symposia. 
Industry Partnership Models 

1. Vertically integrated, demonstration focused- Develop and demonstrate individual 
technologies working with an industry team that represents the "supply chain 
hierarchy." The team works according to a single plan intended to create an 
integrated "aircraft". The plan is agreed to by a standardized "industrial 
cooperative agreement", led by a "prime" integrator, and overseen by NASA 
managers. Knowledge is disseminated informally between team members and 
formally through sharing of intellectual property rights. 

2. Horizontally organized, standards focused- Develop common technology 
standards and systems architectures with broad based industry teams that 
represent competitors within multiple industry sectors. The teams work according 
to mutually agreed upon technical plans. NASA manages the partnership toward 
strategically agreed upon targets. Knowledge is disseminated informally between 
team members working project work, and formally by jointly proposing standards. 
The intellectual property behind new standards is jointly held and shared among 
all team members for a limited period of time and then released to the public. 
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Program Management Strategic Issues: Four Issues Limit Management’s Options 

The AGATE program planning managers also faced a number of unique strategic 
issues in formulating the program plan. These issues required the managers to determine if 
and where traditional program management models would have to be changed. 8 

The first issue was NASA's lack of general aviation focused research capability as a 
result of a decision in the mid-1980's to de-emphasize such work. This had resulted in a lack 
of facilities access and personnel in areas such as computer simulation capability, engine test 
cells, and material property labs. The career interests of NASA personnel reflected this 
previous policy change, resulting in a scattered network of researchers interested in general 
aviation work, but no critical mass capable of undertaking a comprehensive program. 

The second issue was the need to make a wide industry impact, without providing 
preference to any a small group of companies. Within this issue was the challenge of how to 
incorporate new, startup general aviation companies with innovative product ideas, but 
lacking the capital and track record to complete the certification process. 

The third issue was the size of funding, which was capped at $69 million over an 
eight (8) year period. The size of funding relative to the breadth of desired technical 
standards meant the program would have to make difficult tradeoffs and find some way to 
lever small investments into industry wide impacts. 

The fourth issue was the gradual shift in expectations for federal program 
management and performance, which was translating into changing expectations of the 
effectiveness of the program. These changing expectations were the result of multiple 
government management reform initiatives: the Reagan Administration initiative focusing 
on Government reforms to cut bureaucracy and promote international competitiveness 
(Grace Commission), the Bush Administration actions to bring about performance based 
budgeting through the Government Performance Results Act (GPRA), and the Clinton 
Administration initiative to Re-invent Government primarily through force reduction, 
procurement reform, and some degree of new authorities. 9 The trend of changes facing the 
program managers is summarized in Figure 2. 


Figure 2: Shift in Federal Program Principles Over Four Administrations 


Category 

Previous Principle 

New Principle 

Activity Purpose 

Process Based 

Performance Based 

Activity Measure 

Volume of Activity 

Relevance to User 

Timeliness 

Timeliness (Speed) Secondary to 
Fairness and Thoroughness 

Timeliness (Speed) Equally Important 
as Fairness and Thoroughness 

Accountability 

Internally Defined Measures 

Externally Defined Measures and 
Relevancy 

Cost 

Based on Budgets and Federal 
Market Cost Comparisons 

Based on Unit Efficiency and Private 
Sector Market Cost Comparisons 

Flexibility 

Changes Permitted Only within an 
Existing Rules Based System 

Changes Created As Necessary to 
Achieve Objectives of Program and 
Needs of Users 

Benefit 

Public Benefit Achieved by Open 
Dissemination of Information and 
Winner-Take- All Contracts 

Public Benefit Achieved by 
Dissemination with American 
Preference and without Winner-Take- 
All Control of Results 


8 Advanced Subsonic Program Element Plan, NASA Langley Research Center, Hampton, Virginia, 
July, 1993 

9 5th Year Report on Re-Inventing Government, op. cit. 
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Horizontal Partnership Model: Closest Fit to Objectives and Strategic 
Requirements 


The NASA AGATE Program planning manager chose a partnership model organized 
into six key technical teams focused on standards development. The model was tailored to 
add competitive processes to supplement the collaborative processes. This decision was 
consistent with the GA Task Force's final recommendation, to form some type of 
public /private partnership for the program. 11 ' The decision also addressed the strategic 
issues facing the program managers: 

Lack of Focused Research: Increase Industry Participation and Output 
Relevance thru Tech Transfer- The managers chose close partnering with industry on 
specific technical tasks to increase the level of relevant technical work, thereby increasing 
technology transfer. The technology transfer research indicated that technology transfer 
from standard in-house research and university programs was too slow and often 
disconnected from the relevance of user needs. 11 

Lack of Resources: Require Industry Cost Sharing and Link in Other 
Federal Programs- The program was funded at the equivalent of $8 million a year and 
faced an industry whose research and technology base was continually shrinking. The 
program required 50-50 cost sharing of all research tasks with private sector members. The 
managers sought resource-sharing projects with other NASA programs, the FAA, industry 
and academia. 12 Such arrangements would increase the speed of adoption if they involved 
customers and end-users. 

Influenced Industry Wide Standards: Create Public Benefit Thru Standards 
without Private Preference- The managers sought to raise the research and development 
capabilities of the General Aviation industry without giving preference to a particular 
company, investment group or technology. The managers also sought to avoid the 
experience of a "winner-take-all" result from competition for a demonstration plane. 

Shifts in Federal Priorities and Management Policy: Build in a Constant 
Change System- The managers anticipated changes in federal policy, GA research 
priorities, and NASA management practices. They also anticipated that research would 
prove some paths unworthy of further investments, and open new paths for technology 
development. The managers sought to build in change at all levels: strategic, management 
methods, systems engineering and technical research. The managers sought change at a 
level where they could balance collaborative and competitive processes according to the 
need of the technical work. 

The result of the Manager's choice was the formation of an alliance including 
multiple NASA centers and units, the FAA, academia and industry all coordinated through a 
form of centralized alliance functions. The fundamental concept and structure of a broad de- 
centralized collaboration was not new to NASA, but the specific structures, processes, and 
business terms were unique in the Agency's recent history. Given that uniqueness, the 
Managers planned a post-project assessment of the effectiveness of this particular program 
management model. 


10 General Aviation Task Force Report, pg. 47, op. cit. 

11 Paul Masson, Meeting National Priorities: NASA’s Mission to America, Pgs. 4-8, PMA, San 
Francisco, CA, 1989. 

12 Advanced Subsonic Program Element Plan, op. cit. 
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OBTECTIVE AND SCOPE 
Objective 

The objective of this report is to describe the program management model of the 
AGATE program, and assess its ability to meet a combination of "effectiveness measures" 
from multiple levels of stakeholders. The report focuses on four questions: 

• What basic NASA R&D program model did AGATE follow? " 

• What tailoring was done for the AGATE program model? 

• What criteria were applied to assess the effectiveness of the tailored AGATE 

program model? 

• How well did the AGATE program perform relative to those criteria? 

Scope and Sources 

The definition of NASA technology program management models was drawn from 
the NASA (Headquarters and Langley) guidelines on management, program management 
documents, and interviews with line managers. Primary emphasis was placed on NASA's 
formal guidance for program and project management: NASA Policy Guidance 7120.5 (A 
&B) Program and Project Management Processes and Requirements. Interviews were 
conducted with NASA managers to define the status of NASA program planning between 

1992 and 1995. 

The sources of information about the AGATE Program design and business model 
objectives were drawn AGATE Program plans and management guidelines issued during 
the Program. In addition, the report author participated in the AGATE Program design in 

1993 and its implementation through 2001. 

The definition of fundamental Federal technology program management 
effectiveness criteria was drawn from a review of policy guidelines, program directives, and 
position papers from the Executive and Legislative Branches covering the last four 
Administrations. 

Information about the effectiveness of the AGATE program model was drawn from 
five sources: 1) AGATE Closeout Reports authored primarily by private sector and academic 
participants, 2) interviews with NASA managers and private sector counterparts, involved in 
the alliance management structure, 3) results of annual "commercialization" performance 
surveys conducted from 1996 through 2000, 4) results of annual performance audits 
undertaken by an independent auditor, and 5) the results of an independent NASA OIG 
review of the program. 

The report is designed for a reader with basic familiarity of NASA's aeronautics 
program history, the General Aviation industry, and the fundamental models for technology 
collaboration. 
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METHODOLOGY: R&D PROGRAM MODEL & AGATE TAILORING 


The Basic Model: NASA’s R&D Program Model 

The AGATE program was designed on the NASA program model operating in the 
early 1990's. That basic "model" included a practice that provided for "tailoring" a program 
to the particular mission needs. 13 Based on that permission for tailoring, the AGATE 
program design was composed of three parts: 

Policy Level- The creation of objectives and accompanying structures to assure the 
program met national needs and was linked into the Agency's strategy 

Management Level- The conformance to NASA's program management preparation 
processes and creating systems to assure the program can be managed effectively. 

Tailoring for Partnering- The tailoring of program planning processes and 
management systems to address partnering issues. The AGATE program tailoring followed 
a systemized process created for structured program partnerships, and supported through a 
standing NASA commercialization program known as the Joint Sponsored Research 
Program. (See Tailoring and JSR Program) 14 

The Policy Level: Clear Need. Strategy. Management and Results 

The AGATE Program managers addressed policy issues working with NASA senior 
management and the General Aviation Task Force. 

Federal Purpose: Multiple Reasons to Revitalize Technology Development Process - The 
AGATE program was initially launched to support Federal policy goals of improving U.S. 
international competitiveness. 15 The policy basis for improving U.S. competitiveness was 
developed by the Grace Commission during the Reagan administration. The GA Task Force 
identified additional policy needs for improving industry R&D, including anticipated 
improvements to aircraft safety, enhancement of noise control, and creation of the potential 
for new transportation systems. The research and concept formulation of such a new 
transportation systems was completed within the first two years of the program. 16 

NASA Strategy Fit- The AGATE program fit within NASA's Advanced Subsonic 
Program as a minor element. While the strategic fit was logical within the purpose of 
NASA's subsonic research, it did not have a strong strategic fit with other programs 
addressing large-scale aircraft. However, NASA was the only entity within the country with 
the critical mass and capability to re-create R&D links between the General Aviation 
industry and the research community. 

Management- The Task Force concluded that the program should be managed as a 
"partnership" between NASA, FAA, industry and academia. NASA was to act as the senior 
partner, and responsibility and authority for final decisions on most critical issues. 17 

Results- The Task Force and planning managers agreed that the results should be 
tangible, but not targeted to the benefit of any one company. Furthermore, they agreed that 
the results could not act as a "subsidy" to the general aviation industry but rather solve the 
structural problems that impeded current R&D from flowing into the industry. 

Management Level: Follow NASA's R&D Program Model 

The AGATE Program followed the basic NASA management model in place 
in the early 1990' s, but at a formal level designated as a "project" rather than a "program." 


13 Program/Project Management, NASA Management Instruction (NMI) 7120.4 per interviews with 
John Newcomb and Howard Robbins. 

14 Joint Sponsored Research, Program Information Package (PIP), reference NASA NPD 

15 GA Task Force Report, Page 12-13, op. cit. 

16 Small Aircraft Transportation System Concept for the AGATE Project, AGATE Workpackage 12, 
Report 12-002, July 1996. 

17 GA Task Force Report, pages 44-51, op. cit. 
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This designation has significance in the level of preparation and systemization for a given 
activity within NASA. Programs are higher-level groups of activities within organizational 
units known as Enterprises. Projects are smaller scale groups of activities within programs. 
The early 1990's model required both programs and projects to have clearly defined goals, 
objectives, requirements, and financial commitments. So, while AGATE was referred to as a 
"program", it actually was planned and initially operated under the "project" structure of 
NASA's system. The project structure permitted the program to be broader in its plans and 
leave more room for future tailoring. Both programs and projects, however, must complete 
the same basic series of planning elements. 18 

Process and Functional Requirements 

■ Program Formulation - The AGATE program formulation was prepared first by NASA 
managers, and subsequently in collaboration with a cross section of industry and 
academia that agreed to "partner" with NASA in the targeted technical areas. Program 
formulation addressed issues including program planning, systems analysis, technology 
readiness, commercialization, business opportunities, infrastructure, and knowledge 
capture /technology transfer. 

■ Program Approval- The program was reviewed and approved within NASA as an 
element within the Advanced Subsonic program. Its "readiness" to proceed was 
determined to be the readiness of industry partners prepared to collaborate with NASA. 

19 Once that readiness was deemed acceptable, NASA management approved the 
program. 

■ Program Implementation- The program execution was originally designed to operate on 
a centralized basis through the General Aviation Program Office (GAPO) with operations 
support of the Program Controls Branch (PCB). Program implementation support 
procedures were to be designed by PCB based on the Langley Management System 
(LMS). This method of implementation was chosen in lieu of a JSR proposed method that 
would have generated a Program implementation "manual" designed to provide 
detailed guidance about the contents of the customized legal agreement driving the 
partnership, rather than the LMS. A scaled down version of this JSR "implementation 
manual" was generated as the Business Operating Handbook. (See Tailoring) 

■ Program Evaluation- The program evaluation evolved in two stages. The first stage 
involved application of NASA's internal program milestone management and review 
process. The second stage evolved through an annual "measurement" of the program 
effectiveness in terms of its ability to generate research that would lead to industry 
standards to support commercialization. 

Management Requirements 

■ Human Capital - The mix of human capital was the most difficult issue faced by the 
program. The absence of previous General Aviation work at Langley left a dearth of 
individuals with both interest and knowledge. The R&T competency managers received 
feedback that GA work was not the "type" that attracted their personnel. 21 ' The mix 
eventually was solved by securing NASA and FAA personnel to lead teams of private 
sector partners organized into one of six technical areas. In addition, contractors were 
used to support Program Management functions. Finally, a separate individual was 
placed as a "facilitator" working with both the Government and industry partners to act 
as a "good faith intermediary." 


18 NASA has clear policy for this model (NASA Program/Project Management (NPD 7120.4B) and 
Project Management Process and Requirements (NPG 7120.5B)) and management development 
programs (NASA Academy of Program and Project Leadership-APPL). 

19 See milestone #1 on AGATE Program Plan, 1994 

20 Interview with Doug Dwoyer 
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■ Resources Management- The resources were planned and managed through a top-down, 
and bottom up process. Six separate technical teams prepared annual "resource plans" 
which were presented bottom up to a Executive Council. The Council matched the 
bottom up requests against top-down program budgets available for the year. A final 
budget was then proposed by the Executive Council to NASA reflecting that year's 
priorities. Incorporated in this process was life cycle cost (LCC) management and 
program accounting. AGATE program required all research tasks would be "cost 
shared", with 50% from the private sector partner and 50% from NASA. NASA used a 
customized monthly financial report to track use of both NASA and private sector 
resources committed to a given ask. 

■ Risk Management- The program faced a limited number of scientific and operational 
risks. However, it did face some "performance" and "flight safety" risks. (See below) 

■ Performance Management- The program created annual technical plans with detailed 
tasks, milestones and deliverables. These were combined into a master performance 
management plan, which evolved into a masterwork schedule. Authority to modify the 
schedule was in the hands of the appropriate Federal manager, depending on the level of 
milestone to be achieved. 

■ Acquisition Management- Acquisition management was not applicable in the traditional 
sense, since NASA was not acquiring goods and services, but rather cost sharing with 
private sector partners. The majority of the program resources were deployed through 
funded Space Act Agreements, with a minority through traditional contractor structures 
to support Program Management. 21 

■ Safety and Environmental Management - The program faced safety risks with flights, but 
no environmental risks. The private sector partners, using their own aircraft and 
facilities, however, undertook all flights. NASA's standard Safety Review Board structure 
and process was modified to recognize that the risks were borne by private sector 
members only. A new safety Board was created under the coordination of a NASA civil 
servant and composed of representatives from the companies responsible for executing 
individual flights. This "blended" approach evolved into the Executive Safety Review 
Board (ESRB) with policies regarding airworthiness and flight safety procedures for the 
program. 22 

• Security, Export Control and Foreign Involvement- The program had no overt security 
issues. Export control was executed by requirements of partners within the business 
agreement to control transfer of technology abroad. Foreign involvement in the project 
was prohibited by terms of the partnership agreement. 23 

Tailoring for Partnering 

The early 1990's NASA practice encouraged managers to "tailor" their program plans 
to meet their particular mission needs. This practice later became codified as an explicit 
element of planning, when NASA updated its formal guidance for program planning and 
included "tailoring" as one of nine themes to guide NASA program managers. The core 
objective of tailoring was to maximize the ability organize and allocate resources as needed. 

24 While the NASA Program Manager, Bruce Holmes, had determined to tailor the program 
plan to incorporate "partnerships", he was left with the question of the right "partnership 
model" as the beginning point for such project formulation. 

Partnership Based Models: Joint Sponsored Research Program 

The Associate Administrator (AA) for Aeronautics chose the specific form of 
AGATE's partnership model for the Program Manager in 1993. The AA made a decision 


21 Short-Haul General Aviation/Commuter Element, Level II Program Plan, Sec. 9.6, January 1994 

22 AGATE, Executive Safety Review Board, Operating Procedures and Guidelines, November 1996 

23 AGATE JSR Agreement, Section 20.01 and 20.02, limitations on foreign access and 
Commercialization Certifications 

24 Interview with John Newcomb regarding evolution of 7120.4 and 7120.5 
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launch three new programs utilizing partnership models being developed by the 
Commercialization Unit of NASA. 25 The partnership models were based on the R&D 
partnership models extensively used throughout the private sector, and organized in a 
formal program known as the Joint Sponsored Research Program (JSR). That program had a 
series of structured steps that copied the private sector method for creating an "operating" 
research and development partnership. An operating r&d partnership contains partners that 
agree to share costs of developing technology that has a direct interest to every member. The 
partnership is organized as a "project" with multiple levels of responsibility and control. All 
participants are required to be at some form of financial risk, and agree to equally share the 
intellectual property outcomes. The management of the partnership is vested with a third 
party "facilitator" that has no direct interest in the technology. The JSR model was based on 
use of Space Act Authority to provide the customization needed for each program or project. 

The JSR program planning requirements follow essentially the same steps as the 
NASA practice for program planning, but tailored to add the additional steps and issues 
relevant to partnering. The additional partnering steps fell in three basic categories: 

■ Needs Assessment: Partnership Value 

A needs assessment was first conducted to assure the program meet the criteria for a 
good partnership. At least half of the potential "partnerships" referred to the NASA JSR 
program staff were attempts to avoid the procurement system or seek funding for programs 
scheduled for closure. 26 The needs assessment asked basic questions such as whether the 
NASA manager had a clear plan with objectives that could be used to solicit partners, and 
whether those partners had compatible financial, operational and cultural practices. 27 

■ Process Modifications: Steps Particular to Program Partnering 

A program partnership follows steps based on the approach that partners "plan and 
implement" together. This approach requires NASA fully prepare its own plan before 
engaging partners in a process that effectively modifies that plan for joint approval and 
implementation. This practice is common to program planning, except for additional steps 
in which partners are expected to commit resources to successfully implement the plan. The 
key elements of this modified process are: 

• Preliminary Project Planning- Prepare a preliminary program plan that specifies goals, 
objectives, tasks, resources, outputs, organization and management processes. Use the 
preliminary plan to identify where and how partners can contribute to the 

program /project objectives. The planning identifies the mutual benefit to the other 
partners from potential involvement. 

• Partner Solicitation & Selection- Use the preliminary project plan to solicit potential 
partners for mutual planning and information exchange sessions. Select potential 
partners based on clearly disclosed and consistently applied criteria. 

• Joint Modification and Project Implementation - Execute the agreed upon program plan 
according to the agreement under the structures developed during the planning and 
structure stage. This stage includes an ongoing "evaluation" and change process from 
within the partnership. 

• Closeout, Distribution and Conversion- Execute the program closeout according to a 
"closeout" element of the joint plan. 

■ Management Requirements: Structures and Controls Particular to Partnering 

Project partnerships have the same basic management requirements as command and 
control based projects, but with modifications to provide for the risk and reward sharing 
agreement. Most of the management activities are structured as joint, with the senior partner 
having final authority to determine direction of the project. In the case of AGATE, NASA 
was that "senior" partner. Beyond the joint management activities, there are only four 
differences between a traditional program structure and a partnership-based structure. 


23 Three programs directed to utilize partnerships: AGATE, ERAST and RITA 

26 JSR Program Research Results prepared by Paul Masson 

27 JSR Program “Intake and Evaluation” form, prepared 1993 
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• Partnership Management- The structures and processes for joint-direction and 
management of the program. The level of participation and influence is normally 
tied to the level of resources, risks and rewards allocated among partners. The 
elements of partnership management include: 

o Strategic Governance 
o Management Organization and Processes 
o Technical Team Organization and Processes 

• Output Distribution Management- The management of systems and procedures 
to capture outputs (hardware, software, patents, copyrights, etc) for protection 
and distribution among partners. 

• Facilitation and Dispute Resolution - The use of some form of independent 
facilitation and dispute resolution to quickly resolve problems. 

• Change Process- The governance and management systems to implement change 
within the distributed partnership structure. 
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Standard vs. Partnership Based R&D Programs Models 

The summary comparison of the traditional NASA program models vs. a 
partnership-based model shows them to be fundamentally the same in terms of steps, and 
management elements. 

• Policy- Both models address policy issues. 

• Assessment- The partnership model includes a preliminary "assessment". 

• Process Requirements- The partnership model includes an additional three process 
elements; two designed to include partners in the planning process, and one designed to 
closeout the partnership in an orderly manner. 

• Management Requirements- The partnership model includes four additional 
management elements designed to deal with the requirements of joint management 
processes. 

Figure 3: Standard vs. Partnership Based Program Model 
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Partnership Based Program Model 

The AGATE program model covered all of the basic NASA program requirements at 
the project level, modified with a number of steps necessary for partnering success. 

Figure 4: Partnership Based Program Model 
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Six steps to assess value of partnership based 
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Ongoing identification of changes with 
concurrent realignment or resources and 
lower-level teaming structure 
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AGATE Organizational Structure 

The AGATE organizational structure was a hybrid of basic project command and 
control structures, with layers of working groups to coordinate decisions. NASA took the 
leadership role at each level, except for basic facilitation and business administration, while 
industry took a "membership" role at each level. The organization had six basic components: 

■ General Aviation Program Office (GAPO)- The NASA program office staffed by the 
Program manager, a deputy and secretary. GAPO was the overall "alliance" manager 
and had final authority to approve all plans. 

■ Executive Council- A senior level council composed of elected representatives from 
industry and designated government representatives. The Council's primary function 
was to agree on strategy and priorities. 

■ AGATE Management Team- The Management Team's primary function was to assure 
coordinated implementation of the annual plan. The team was composed of the GAPO 
personnel and government technical specialists that led six technical working teams. 

■ Systems Assurance- The systems assurance team was led by a Government specialist and 
composed of industry members representing each of the technical working teams. Its 
primary function was to support systems integration across technical teams and assures 
work was done to overall "configuration management." 

Figure 5 

AGATE Alliance Organizational Structure 



Legend: Green lines indicate "membership participation" 
Black lines indicate "leadership" responsibility 
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■ Technical Teams- A total of six technical teams were formed. Each team was led by 
Government specialists and composed of industry and academic members. These were 
the basic organizational units that prepared and executed annual technical plans to 
achieve the program's objectives. 

■ Facilitation/ Business Administration- The facilitation function acted as a good-faith 
intermediary between the public and private sector members. The business 
administration function provided eight basic services necessary for a partnership-based 
program. Led by private sector managers, this function was delivered through by a non- 
profit private sector entity, and funded by both the public and private sector. 
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EFFECTIVENESS MEASURES: AGATE PROGRAM CONTEXT 


Four Levels of Effectiveness Measurement 

There were four levels of effectiveness that applied to the AGATE program over its 
lifetime of planning and implementation : 28 

• Policy- Major principles that guide the justification all Federal programs. 

• Department /Agency Strategy- Strategic goals to be accomplished or supplemented by all 
Agency programs . 29 

• Large Organizational Unit- NASA Center (Langley, Ames, Glenn) level measures 
including their respective charters, developmental goals, and management system 
requirements. 

• Program- Measures specific to a specific program (e.g. aeronautics knowledge 
advancement). The achievement of the targeted program outcomes must meet 
effectiveness requirements at each of the three higher levels. 

Eight Measures of Effectiveness During AGATE Program 

There has been a concerted effort across multiple administrations to continuously 
improve the operating performance of the Federal government. A review of the 
management initiatives launched over the past four administrations was synthesized into 
eight major effectiveness criteria, which were applied to the AGATE program . 30 These 
measures translate into specific management expectations of stakeholders at different levels 
of a given Federal program. 

• Performance- Creation of measurable goals to be achieved through program plans, 
organization structures, and management structures with high quality, on schedule and 
within budget. 

• Public Benefit without Private Preference - Creation of Program outputs designed to 
generate benefits to the American public without subsidy to private participants or 
control of the outcome by a limited number of organizations. 

• Relevance- Definition of goals and objectives relevant to customer /user groups. 

• Speed- Generation of results quickly and in forms that can be readily adopted by 
customer /users. 

• Accountability- Creation organizational structures with clear responsibility and 
monitoring systems tied to specific performance measurements. The structures and 
systems hold Program and task level managers accountable for all Levels of objectives. 

• Cost- Reduction of costs in all categories of program content and management. 

• Flexibility- Use of legal authorities, organizational structures, and decision processes to 
respond to changing needs from policy, management, and research & development 
finding. 

• Institutional Development- Development of institutional capacity for further research 
and development support. Development of these capacities includes improved human 
capital, facilities, and new knowledge 


28 Derived from interviews with Bruce Holmes, Sam Morello, and David Trinkle (OMB) 

29 NASA Strategic Management Handbook, as revised NPG 1000.2, Washington, DC, 2000 

30 A Report to the 43 rd President and 107 th Congress: Transitioning to Performance-based Government, 
November 2000 
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Assessment Framework 

The cumulative program effectiveness measures generated at different levels of 
oversight create a single model for the AGATE program assessment. The format is designed 
to mirror the evolution in performance measurement summaries developed under the last 
two administrations . 31 32 


Figure 6: Assessment Framework 
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Assessment Rating 

The effectiveness feedback is summarized qualitatively at one of three levels (high, 
medium and low). The qualitative rating was compiled in from the five sources identified in 
the Methodology. Performance and Accountability assessments were heavily based on 
interviews from NASA managers 33 , annual "commercialization" performance reports, 
annual performance audits, and the independent OIG review. Relevance, Cost and Speed 
assessments were heavily based on closeout reports and annual "surveys" conducted of both 
industry members and NASA technical team leaders. Benefit Without Preference was 
heavily based on the annual commercialization report. Flexibility and Institutional 
assessments were heavily based on the interviews with NASA technical team leaders and 
business managers. 


31 5th Year Report on Re-Inventing Government, op. cit. 

32 Program Assessment Rating Tool (PART) explanation, provided by David Trinkle to Paul Masson 
for interview. 

33 See Appendix A for interview guide 
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EFFECTIVENESS: PARTICIPANT CONCLUSIONS 
Summary 

The participants rated AGATE as achieving its primary objectives and rating well on 
effectiveness in most areas, with high measures for relevance, cost, speed and public benefit, 
but lower measures for institutional fit and flexibility at dealing with the larger NASA 
organizational structure. This pattern mirrors private sector surveys and represents a 
tradeoff between the benefits of tailoring a program using partnering, versus the changes 
necessary within the institutional structure to support such tailoring. 

The technical managers generally rated the effectiveness high when assessed against 
the program's stated goals. The area of stretch goals is where the technical participants rated 
AGATE lower. Technical participants sought more integrated flight tests, but recognized 
that integration was a tradeoff for a desired "decentralized" approach permitting work 
teams to operate independently on selecting specific technology targets. 

The business managers at NASA rated the program medium to low, focusing on the 
difficulty of implementing a program structure that contained more tailoring than most 
NASA LaRC programs. 

The summary of effectiveness responses is provided in the following table. 
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Figure 7 : Summary of Effectiveness Assessments 


MEASURES 

Qualitative 

Rating 

Effectiveness at Policy, Strategic, Organizational 
Unit and Program/Project Levels 

Performance 

High 

Performance was achieved at all levels. The 
decentralized structure of the alliance made it difficult 
to achieve integrated flight tests. 

Relevance 

High 

The relevance of program/project outputs scored high 
by the industry end-users and the FAA partners. 
Disagreement over relevance was limited to specific 
technology targets rather than the overall 
program/project approach. 

Benefit w/out 
Preference 

High 

Public benefit without private preference scored high 
with all participants due to the focus on creation of 
technology standards rather than specific technologies. 

Accountability 

Medium 

Accountability was achieved at the Policy, Strategic 
Levels. Financial accountability was achieved at the 
Center and Program level, but technical accountability 
was reported as inconsistent for some of the work- 
teams at the program/project level. This inconsistency 
at the program/project level created questions of 
accountability at the Center level. 

Cost 

High 

The cost effectiveness was considered high at all 
levels. The program secured cost sharing from private 
sector partners and organized highly effective 
leveraging agreements with other NASA programs 
(SBIR) and the FAA. 

Speed 

High 

The speed of tasking and undertaking work was 
considered high, when compared to other Federal 
program/project forms. The speed of program 
“formulation” was the only negative rating. This was 
attributed to the need to secure “agreement” with 
partners before proceeding. 

Flexibility 

Medium 

Flexibility was achieved at the Policy, Strategic and 
Program/Project levels. Changes in national priorities, 
NASA’s strategy and project targets were 
incorporated into the program. This flexibility, 
however, caused difficulties at the Center level need 
for consistent program/project management. 

Institutional 

Low 

The program worked with a limited section of the 
Center level institutional structures and built 
certification rather than research knowledge in the 
research and technology competencies. Center level 
business managers found the program required 
difficult modifications from standards procedures. 
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Performance 

The program achieved its performance goals at all of the four measurement levels. 
NASA performance goals are officially classified using similar terminology that denominates 
"levels" 1 through 3 for performance, with level 1 representing policy and strategic goals, 
level 2 representing organizational unit goals and level 3 specific program /project 
performance measures. 

The Policy and Strategic goals were achieved by the re-creation of links between the 
aeronautics research and development capabilities of the country (NASA, FAA and 
academia) and the private sector general aviation industry. Commercial and academic 
participants in an annual commercialization survey confirmed ongoing research links based 
on one of fourteen "standards" developed by AGATE program. 34 Examples of such work 
include avionics advances for additional pilot information providing new safety; adoption of 
noise prediction software permitting better design to minimize noise; creation of new 
methods for composite materials use that is now being adopted by both large aircraft 
manufacturers and the military. 

Policy and Strategic unit goals were also achieved by changing technical work plans 
and priorities. The changes were possible due to the organizational structure of the program 
that permitted adding and dropping technical teams and creating special "task element" 
teams by mutual agreement among the members. Three examples of this shift were: 

Security- The addition of free flight tests for the FAA over the Atlanta Olympics in 
response to the need for increased security. These tests were accomplished by having the 
FAA sign into the agreement as an AGATE member; fund a separate task within a 
technical team; and secure cost sharing from the private sector to create the "standards" 
behind the technology necessary to support free flight. 35 

Safety- The creation of an aviation safety initiative resulted in the funding by NASA's 
aviation safety program of a new team within AGATE addressing safety issues specific to 
the general aviation industry. 

Noise- The increase in concern regarding noise resulted in the creation of another task 
element team with the Integrated Design and Manufacturing technical team focused 
solely on noise prediction and limitation within general aviation aircraft. The resulting 
software code has been adopted and is portable across multiple platforms, and was 
reported by AGATE members as also being applied to commuter aircraft design. 

The program met its objectives at the Organizational Unit and Program level (Level 3), 
after in Executive Council and the Program managers. These Level objectives were grouped 
by technical area, with the largest number of objectives met in the area of integrated design 
and manufacturing followed by flight systems. The area that saw the least number of 
objectives met was in integrated flight platforms. One of the "stretch" goals of the program 
was to integrate advances from the teams into aircraft for flight demonstrations. For 
planning purposes, the evolution of technologies was grouped sequentially into a 
"reference" point designated reference aircraft "A", "B" and "C". The program did not 
succeed in securing sufficient integration of technologies to successfully launch all of the 
reference aircraft. In this sense, the AGATE model, with its decentralization and separation 
of tasks, was not effective as delivering demonstration flights incorporating integrated 
technology. 


34 Keith Gale, AGATE Commercialization Report, 1995-2000 

35 Free flight is the ability of aircraft to operate safely in regulated air space without the need for 
a central air traffic control. 
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Relevance 

The Program received high ratings for the relevance of its work from industry and 
user participants based on an annual survey. The AGATE program conducted an annual 
survey among members of the most value received from participation. The highest value 
ranked by the members was the "organizational" structure of the program, which brought 
together the diverse elements of aeronautics research in the country focused on the needs of 
the General Aviation aircraft. Financial value ranked at the bottom of the survey, reflecting 
the small amount of money often allocated across the tasks. The relevance value was 
concentrated in three areas: 

Focus on End-User Related Standards- The program focused on generating 
Guidelines for Standards and Certification (GS&C). The relevance of these standards was 
assured by the open participation by any US general aviation company. The requirement for 
cost sharing among participants assured that only those "standards" broadly supported by 
the industry were undertaken during the program. The participation by these stakeholders 
in defining the technical plans, tasks and deliverables assured they were relevant to 
members. 

Tom Bond of the Glenn Research Center led work in Icing Systems and stated:" This 
vehicle offered more potential to work with the end user than any other vehicle I have ever 
used. We became more closely coupled to the final state of our technology use, and I found 
that very valuable.... It serves an excellent purpose for NASA because it balances our work: a 
certain part into the next generation, a certain part into the near term exploratory, and a 
certain part in the 3 to 5 year impact." 36 

Link to FAA Certification Requirements- The legal authority and organizational 
structure of AGATE permitted a series of technical exchange links between AGATE 
members and the FAA. These links were judged by representatives from NASA, the FAA 
and members to be one of the most effective features of the AGATE program. The link took 
the form of a working group of Federal Aviation Administration (FAA) certification 
personnel from around the country, organized by their respective disciplines matching the 
field of technical research within AGATE. The group was known as "AIR AGATE" and met 
collectively with industry technical counterparts within AGATE to review the type and 
quality of information necessary to establish standards that could support possible 
certification guidelines. The creation of a national group provided for an increased level of 
consistency when applying such standards and certification guidelines by the FAA. The 
ability of industry specialists to collectively discuss technical standards with the FAA 
permitted private corporations to effectively plan product innovations for review according 
to FAA expectations. This, in turn, resulted in lower certification costs and better 
conformance to FAA certification regulations. 

Private Sector Investment- The program eventually attracted more private sector 
investment than originally required. The program was designed to require dollar for dollar 
matching of research funding. An audit of nine private sector members indicated that five of 
the nine audited members exceeded their obligation to match Federal research funding by a 
collective 19.3%. 37 The private sector members reported the rationale for additional 
investment as the value of the standards for industry growth and certification cost control. 

Public Benefit Without Private Preference 

The program participants and reviewers were in agreement that public benefit was 
achieved without private preference. This was achieved through three mechanisms: 

Creation of Standards for Competition- AGATE research results took the form of 
technical reports designed to generate "pre-competitive standards". The reports, called 


36 Paul Masson interview with Tom Bond 

37 Conclusions From Review of AGATE Audits: Total AGATE Member Resource Matching for 
Project Years 1995-2001, prepared by Paul Masson and Bill Lennett, Contractor Report L-70960D 
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Guidelines for Standards and Certification (GS&C), were made available to any US general 
aviation corporation willing to share in the AGATE program cost of their creation. 

The guidelines acted as the equal playing field for building new products among 
competitors, without benefiting any single company that would control the guideline. 

Organization Permitting Unlimited Participation- The AGATE organization structure 
permitted participation by all sizes and types of organizations. Three classes of membership 
were created: 

■ Principal- Designed for large corporations willing to meet the minimum 50-50 
cost sharing guideline with rights to all AGATE reports. 

■ Associate- Designed for small corporations willing to pay less than the minimum 
50-50 cost sharing, and receive limited rights to AGATE reports. 

■ Supporting- Designed for universities, corporations and individuals unable to 
cost share, but willing to pay a nominal membership fee in return for rights to a 
small number of systems reports. 

The conditions of participation required Principal and Associate members to develop 
commercial product innovations based on AGATE developed standards. 3 * 5 Foreign 
participation was restricted. 

Intellectual Property Terms - The AGATE technical reports (GS&C's) were defined as 
a class of intellectual property known as "data rights." The data rights terms required the 
reports be released to the public in five years, or earlier if agreed to by the members. In 
addition, the property rights on the reports permitted them to be "traded" between the 
various technical teams within the overall AGATE Alliance. 

The technical areas of the reports included, among other items, crashworthiness 
design guides for safer aircraft; avionics certification standards permitting new information 
tools in the aircraft for safer aviation; and composite materials qualification methodologies. 

Accountability 

Accountability was perceived differently by the participants, depending on their 
particular level of responsibility and systems normally used for project monitoring. 

Policy and Strategic level accountability was achieved via regular Program reviews at 
NASA headquarters. This was accomplished through annual reviews of the program with 
concurrent reporting to goals and milestones. 

Center level and Program level accountability received a mixed review. The 
institutional program /project monitoring managers at the Langley Center did not rate the 
accountability high within the program. There was a mix of positive and negative factors 
that contributed to this question of accountability: 

Positive: Clear Technical and Business Administration System - The AGATE program 
created a business administration and monitoring system customized to the particular 
structure of its project. This system was codified in an Administrative Management 
handbook that combined the JSR Agreement, the JSR program guide known as the Program 
Information Package (PIP), and a detailed business management guide known as the 
Business Operating Handbooks (BOH). Within the JSR Agreement appendices were details 
of required annual technical planning, monthly reports, and compliance approval systems. 
The overall administration of the project was assigned to the NASA LaRC Project Controls 
Branch. 

Positive: Regular Audits and NASA Technical Review - The program had a built 
annual series of member audits and provisions for special reviews. A total of nine audits 
were completed on members during the length of the program. The audits found no major or 
misrepresentation by members. In addition, the OIG undertook a separate management 
review, resulting in a recommendation to change one operating procedure. 

Negative: Withdrawal of PCB, Creation of New System- The NASA LaRC Project 
Controls Branch (PCB) withdrew support of the program after one year, citing the Federal 


38 AGATE Joint Sponsored Research Agreement, Membership Category, Sec. 4.04 


28 



downsizing and lack of personnel to support the AGATE program. The PCB withdrawal 
required creation of a separate business administration and accountability system. The 
AGATE private sector members created a non-profit corporation to assume the majority of 
PCB's support functions. The non-profit was chartered to support the AGATE Alliance and 
was known as the AGATE Alliance Assn., Inc. (AAAI). This was perceived as losing 
Government monitoring over the project, even though AAAI was required to provide all 
monitoring and review requested by the Government. 

Negative: Inconsistent Technical and Business Reporting Enforcement - The program 
experienced incomplete or late technical and business reporting throughout its technical 
teams. The incomplete reporting required pro-active follow-up by Government technical 
leaders and business administration personnel. The additional follow-up often took the form 
of verbal and e-mail confirmations of information required for program oversight. This 
resulted in a gap between the detail of information maintained in the central administrative 
archive, and that found in files of the technical team leaders. 

Cost 

The AGATE program was completed with substantial contributions from the private 
sector and close coordination with other Federal agencies and small business programs. The 
resulting leveraging generated a program in which NASA program funding paid 49% of the 
total cost over seven years. The program received $69 million over eight years, with $40 
million spent on direct technical research and $29 million systems assurance, program 
management, and alliance management. Of the approximately $140 million total AGATE 
expenditures, $40 million were paid for by the private sector, and another $34 million 
levered from FAA, DOD, and SBIR/STTR programs. 39 

There were multiple mechanisms that made this cost effectiveness possible: 

Program Formulation and Legal Authority - The program was initially formulated to 
operate as a cost-sharing program with the private sector. The use of Space Act Authority 
within the Joint Sponsored Research Program permitted NASA to create "partnership" 
categories with multiple criteria for cost savings. 40 The authority also permitted NASA to 
create participation categories for small businesses to offer internal research in exchange for, 
access to the AGATE technical reports (GS&C's). 

Close Program Coordination: SBIR/STTR - The AGATE program managers created a 
closely coordinated plan between AGATE technical topics and those due for announcement 
in the SBIR/STTR programs. 41 This coordination permitted small businesses to undertake 
research relevant to the topics selected within AGATE. 

Clear Cost Sharing- The program developed a detailed set of cost-sharing conditions 
that went beyond the OMB guidelines. These conditions specified the particular forms, 
levels and criteria for cost sharing. 42 One form of contribution permitted small businesses to 
present proprietary technical reports as an "in-kind" contribution if agreed to by a given 
technical team. In addition, the program sought and received one-time approval to use 
Space Act Authority to permit background intellectual property (IP) to be used as cost 
sharing. However, only three background IP contributions were made, all in small dollar 
amounts. The lack of use of background IP contributions was attributed to the strict criteria, 
which copied private sector IP valuation criteria for such IP. 

Verification Criteria and Process 

The program developed a process for verifying contributions by non-federal 
members. The process required annual submission of planned financial contributions by all 
non-federal members. This submission, known as the annual "in-kind cost-sharing" report. 


9 Total AGATE Program Funding Report, prepared by Tim Warner, NASA LaRC 

40 AGATE JSRA, Membership Category, Sec. 4.04, op. cit. 

41 Interview with Thayer Sheets, NASA LaRC SBIR Program Manager 

42 AGATE JSRA, Allowable Resource Contributions (Attachment 9), In-Kind Resources Valuation 
Methods (Attachment 10), Financial Procedures and Allowable Expenses (Attachment 14), op. cit. 
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was reviewed by the independent facilitator for compliance with the cost-sharing criteria. 
Those companies that were not within compliance or lacked supporting information were 
required to correct their contributions or be subject to loss of membership rights. Companies 
that submitted incomplete or financial information to support their planned in-kind cost 
sharing were subject to one of the annual audits. (See Accountability) 

Speed 

The overall speed of the program was rated high due to the speed of individual task 
planning and implementation. The program was described as "slow" in the formulation 
stage due to lengthy meetings with private sector participants to agree upon specific 
technical plans. Specific areas mentioned in the interviews and closeout reports included: 

Faster Project Startup- Startup activities beyond the planning were faster due to the 
time spent in up-front agreements with members and legal authority. The JSR Authority 
permitted development of "Annual Task Plans" that were attached to a master agreement. 
The task plans could be initiated once the master agreement had been signed. 

Rapid Strategic Redirection - Strategic re-direction was faster due to the combined 
operation of the Executive Council and the ability to change task statements without re- 
negotiating the master partnership agreement. 

Slow Initial Program Formulation- The participant structure and motivation coupled 
with the team process emphasizing consensus reduced the speed of program planning and 
startup, except in the area of procurement execution. Overall, the program was slower to 
create plans and launch than a comparable government planned program. 

Task Execution and Outputs - The legal authority and participant motivation of 
AGATE increased its effectiveness at task execution and the ability to adjust when necessary. 
The legal authority permitted the creation of a task funding process that was completed 
within one-half the normal time of procurement. The participant motivation of AGATE, with 
its cost-sharing feature, was an incentive to finish tasks on time and budget, since the cost of 
extensions and overage were equally born by both the Government and private member. 

Flexibility 

The program received positive reviews for its strategic and operational level 
flexibility, but that flexibility caused difficulties working within the Center level 
management systems. 

Change Process - The key to change was in the terms of the legal agreement, which 
permitted the program to be re-defined and re-tasked every year. The agreement was 
structured to address long-term research collaboration issues, rather than a single research 
and development program for a fixed period of time. The changes were executed by mutual 
agreement through procedures established for the Executive Council and Technical Teams . 43 
Differences over direction of change were resolved by involvement of the Facilitator. 44 

Competitive vs. Collaborative Processes - The legal agreement provided for the 
introduction of competitive processes when the basic collaborative could not produce 
agreement. The competitive process was defined by NASA following modified procurement 
procedures and was known as a Competitive Tasking Notice (CTN). The process was 
invoked twice, with split agreement among members as to the success of the process. 

Separate Technical Teams with Ability to Trade - The Technical Teams constituted the 
core organizational unit for all technical work. The intellectual property rights of the legal 
agreement provided that the results of each team were "separate" from other teams. 
Elowever, the legal agreement also permitted the teams to "collectively" trade information 
and exchange intellectual property rights, thereby creating a form of internal market for 


43 JSRA, Executive Council Operating Procedures (Attachment 3), Technical Council Operating 
Procedures (Attachment 4). 

44 JSRA Legal Agreement, Alliance Facilitation (Attachment 5). 
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exchange. In this manner, the teams were flexible in judging what was and was not of value 
in creating integrated systems. 

Flexibility vs. Program Management Structures - The highly codified procedures for 
the change processes and working teams translated into inflexibility at the local Center level. 
It became difficult for Center level management to institute changes in management systems 
and bring the AGATE Program in alignment with those changes. (See Institutional below) 

Institutional Fit and Development 

The program received mixed or negative assessments from Center level managers 
responsible for assuring the program fit within overall institutional standards and structures. 
Most of the individuals interviewed understood that the program was intentionally designed 
to operate differently. However, that understanding did not address their particular issues, 
which generally fell into three categories: 

Center Level Management and Business System Fit - The program was initially 
designed and launched without developing some of the standard elements of 
program /project management evolving at the Langley Center in the early 1990's. These 
elements, known collectively as the Langley Management System (LMS) called for certain 
minimal criteria in program /project design and implementation. Some of those elements 
addressed issues such as configuration management, technical output control, acquisition 
management, safety assurance, and logistics management. Those elements were not 
explicitly addressed in the AGATE program during its formulation. The core conflict stems 
from AGATE's design of "processes" to create ongoing technical programs rather than a 
fixed research and development program. 45 The secondary conflict stemmed from the use of 
Space Act Authority to create a task financing process that was a modification of cooperative 
agreements, but without the documentation desired by procurement managers. One LaRC 
procurement manager described AGATE as "...not the way to do business." 

Center Level R&T Competency Relevancy - The AGATE model of strategic planning, 
task staffing and technical output definition did not generate results relevant to the R&T 
competencies within NASA Langley. The technical strategy of AGATE was focused on re- 
building general aviation advanced technology links between government, industry and 
academia. This very objective limited the creation of new knowledge that would have been 
relevant to the NASA R&T competencies. In addition, the General Aviation category was 
not a high priority with individual researchers. "The feedback I got was that the research 
was not the kind... they (the researchers) wanted to do and that (the researchers) were 
expected to led a fractious group of people." 46 

Center Level Capabilities Development: Facilities, Personnel, and Knowledge- The 
program did not create new facilities or knowledge that added to the Center's long-term 
capabilities. The project staffing via participating members rather than Center based 
contractors reduced the close-in knowledge transfer that occurs between government 
contract technical monitors and their contractor counterparts. The definition of technical 
outputs as "guidelines for standards and certification" meant that the information contained 
in those guidelines was rarely new knowledge, but rather the configuration of the correct 
knowledge necessary to deploy a new technology through the FAA certification process. 
Finally, the bulk of the work was undertaken at facilities of corporate and academic 
members, with the exception of drop tests conducted at Langley's facility. 


45 Advanced Subsonic Program Element Plan, op. cit. 

46 Interview with Doug Dwoyer 
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Appendix A- Interviews and Interview Questionnaire 
Purpose and Scope 

Interviews were conducted with individuals at two levels of influence: 

• Policy- Federal and NASA policy regarding science / technology program 
use of public/ private partnerships. 

• AGATE Program Design and Implementation- Participants in the design 
and implementation of the AGATE program. 

Schedule 

Interviews were conducted in three rounds. The first round was focused on 
program designers and participants, and was completed between August and 
November 2002. The second round was focused on NASA managers responsible for 
program & project policy at the Agency-wide and center level, and was completed 
between January and March 2003. The third round was focused on Federal 
managers with influence on the standards for management effectiveness, and was 
completed between April and June 2003. 

Interview Guide(s) 

Interview guides were developed for the two separate groups of interviews. 

Policy 

1. Support Partnership Authorities and Business Practices- What, if any, 
were the reasons that motivated you and your colleagues to support the 
effort to adopt legal authorities and business practices permitting 
federal technology alliances such as AGATE etc.? 

2. Pursue Management Effectiveness- To what extent, if any, was improved 
management effectiveness a motive in supporting the adoption of such 
authorities and business practices. We are defining management 
effectiveness as the ability, as a minimum, to achieve federal R&T program 
goals more 1) quickly, 2) with fewer resources, 3) without political, policy or 
program conflicts, and 4) without burning out the human resources. 

Program Design Implementation 

1. Flow would you compare the design and implementation to 

A. FAR based contract for a demo 

B. University led consortia 

B. Industrial cooperative agreement with a vertically integrated team 

2. What was more effective, if anything, about AGATE? 

3. What was less effective, if anything, about AGATE? 
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Appendix A- (Con't) 


Interviews- Policy and Strategy 

Name Former/ Current Position Policy / Strategy Role 


Ruth Martin 

Director for Program 
Integration 

Link between AGATE Program 
objectives and NASA LaRC 
Aeronautics Program Strategy 

Doug Dwoyer 

NASA LaRC- Assoc 
Director R&T 
Competencies 

Link between supplying R&T 
competency expertise to 
AGATE and measures of R&T 
competencies groups 

Sam Morello 


Link between detailed AGATE 
program objectives and needs of 
both Program Integration and 
R&T Competencies 

Ambassador James 
R. Jones, formerly 
Congressman James 
R. Jones (D-Ok) 

Counsel Manatt, 
Phelps 

Co-Chair Manatt, 
Jones Global 
Strategies, LLC 

Policy regarding NASA charter 
and budget. Chief of Staff 
President L. Johnson (1963-68), 
basis for policy to support 
strategy linking technology 
programs to technology transfer 

Frank Penaranda 

NASA Retired, 
formerly Chief of 
Technology Transfer 
Programs 

Policy regarding technology 
transfer via partnerships. 

Kevin Barquinero 

President, 

Knowledge Sharing 
Systems, formerly 
Technology 
Commercialization® 
NASA HQ 

NASA Joint Sponsored Research 
Partnership Program design and 
implementation; liaison to 
NASA Aeronautics for selection 
of AGATE, ERAST and RITA 
alliances 

John Newcomb 

NASA Consultant 

Develop management 
effectiveness criteria for NASA's 
Advanced Program and Project 
Leadership (APPL) 

Paul Coleman 

University Space 
Research Associates 

Develop strategy for creation of 
institutional partnering 
programs at NASA 

David Trinkle 

Office of 

Management and 
Budget 

Evolution of policy regarding 
measurement of program 
effectiveness, link to creation of 
Program Assessment Rating 
Tool (PART) 
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Appendix A (Con't) 

Interviews- Program Design and Implementation 


Name 

Current Position 

Design & Implement Role 

Bruce Holmes 

NASA LaRC-SATS 
Project Office 

Manager- LaRC General 
Aviation Program Office 
(GAPo)- Federal manager of 
AGATE 

Mike Durham 

NASA LaRC- SATS 
Project Office 

Dpty. Manager- GAPO- 
Deputy manager for AGATE 
1996-1999 

Sally Mauldin 

NASA ARC-Chief 
Counsel 

Lead LaRC attorney on 
drafting of AGATE legal 
agreement 

Steve Hanvey 

President- Piaggio, 
USA 

Leader of AGATE Executive 
Council-1995-1996 

Ed Hooper 

Toyota Motor Sales, 
USA- Aircraft 
Development 

Beech representative to AGATE 
Program planning task force- 
1992-1993 

Sandra Ray 

NASA LaRC- Dpty. 
Director Procurement 

Procurement representative to 
AGATE business management 
design team 

(N-Team) 

Rosemary Froelich 

NASA LaRC- Head 
Grants and R&D 
Studies Branch 

Procurement liaison from 1996 
to 1999 

Karen Reilley 

NASA LaRC- Office of 
Chief Counsel 

GAPO and AGATE program 
government counsel 1997 to 
2001 

Sandra Smalley 

NASA HQ- Financial 
Management 

AGATE Business 
Administration Manager via 
LaRC PCB- 1995 to 1996 

Michelle Cohoon 

NASA LaRC- Financial 
Management 

AGATE Business 
Administration Manager via 
GAPO 1998-1999 

Tim Warner 

NASA LaRC- 
Management Support 
Office 

GAPO Program Analyst for 
AGATE- 1997 to 2001 

Jack Sheehan 

AGATE Alliance 
Association, Inc. 
(AAAI)- ED 1997-2001 

AGATE business 
administration services 

3Q-1997 to 2001 
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Tom Bond 

NASA GRC- Icing 
Branch 

AGATE Icing Protection 
Systems Technical Team Lead 
1995-1998 

Paul Stough 

NASA LaRC- Crew 
Vehicle Integration 
Branch 

AGATE Flight Systems 
Technical Team Lead- 1995- 
1997 

Tom Freeman 

NASA LaRC- 
Mechanics and 
Durability Branch 

AGATE Integrated Design and 
Manufacturing Team Lead- 
1996 to 2001 

Ted Bright 

NASA LaRC 

Executive Safety Review Board 

Walt Green 

NASA LaRC 

Systems Assurance 
Workpackage Leader- 1997- 
2000 

Thayer Sheets 

NASA LaRC- 
Technology 
Commercialization 
Program Office 

NASA SBIR Program Lead at 
LaRC 
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